Image processing apparatus and image processing method

ABSTRACT

An image processing apparatus includes: an interface unit configured to input an image signal from an imaging apparatus that exposes a specimen dyed with a fluorescent dye to excitation light and images fluorescence by a color imaging element; and a color correction circuit configured to retain information on a percentage of each of a component of a second color and a component of a third color with respect to a component of a first color corresponding to the excitation light in the image signal, which is determined in advance based on color filter spectral characteristics of the color imaging element, and reduce each of an amount corresponding to the percentage of the component of the second color and an amount corresponding to the percentage of the component of the third color from the input image signal.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of Japanese Priority PatentApplication JP 2013-239374 filed Nov. 19, 2013, the entire contents ofwhich are incorporated herein by reference.

BACKGROUND

The present technology relates to an image processing apparatus thatprocesses a fluorescence image captured by an imaging apparatus and toan image processing method.

Image processing apparatuses each of which captures an observation imageof a fluorescence microscope by an imaging element and acquires it aselectronic image data are used in the fields of, for example, medicalscience, biological studies, and examinations.

The observation image of the fluorescence microscope is an image of weakfluorescence. It is thus difficult to obtain a contrast (visualdifference) between an image of a cell portion dyed with thefluorescence and an image of a background portion without the cell.

In this context, Japanese Patent Application Laid-open No. 2004-086031describes a technique of black balance correction of increasing thecontrast between the image of the cell portion and the image of thebackground portion by subtracting, from the observation image, aluminance of the background portion manually specified by a user suchthat the background portion is darkened.

In addition, Japanese Patent Application Laid-open No. 2010-098719discloses a technique for automatically performing white balancecorrection or black balance correction without the user manuallyspecifying the background portion.

SUMMARY

In the field of the image processing apparatus that processes afluorescence image obtained by an imaging apparatus such as afluorescence microscope and generates an image for observation, atechnique for generating a fluorescence image easily observable by theuser is becoming more and more important and needs to be developed.

In view of the above-mentioned circumstances, it is desirable to providean image processing apparatus capable of generating a fluorescence imageeasily observable by the user and an image processing method.

According to an embodiment of the present technology, there is providedan image processing apparatus including:

an interface unit configured to input an image signal from an imagingapparatus that exposes a specimen dyed with a fluorescent dye toexcitation light and images fluorescence by a color imaging element; and

a color correction circuit configured to retain information on apercentage of each of a component of a second color and a component of athird color with respect to a component of a first color correspondingto the excitation light in the image signal, which is determined inadvance based on color filter spectral characteristics of the colorimaging element, and reduce each of an amount corresponding to thepercentage of the component of the second color and an amountcorresponding to the percentage of the component of the third color fromthe input image signal.

The color correction circuit may be configured to reduce the componentof the first color from the input image signal.

The color correction circuit may be a linear matrix transformationcircuit.

The image processing apparatus according to the present technology mayfurther include a white balance adjustment circuit configured to reduceeither one of the component of the second color and the component of thethird color from the input image signal at a preceding stage of thecolor correction circuit.

The fluorescent dye may be fluorescein.

According to another embodiment of the present technology, there isprovided an image processing method including:

determining, based on color filter spectral characteristics of a colorimaging element that exposes a specimen dyed with a fluorescent dye toexcitation light and images fluorescence, a percentage of each of acomponent of a second color and a component of a third color withrespect to a component of a first color corresponding to the excitationlight in an image signal of the color imaging element; and

reducing each of an amount corresponding to the percentage of thecomponent of the second color and an amount corresponding to thepercentage of the component of the third color from the image signalinput from the color imaging element.

As described above, according to embodiments of the present technology,it is possible to generate a fluorescence image easily observable by theuser.

These and other objects, features and advantages of the presentdisclosure will become more apparent in light of the following detaileddescription of best mode embodiments thereof, as illustrated in theaccompanying drawings.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a view showing a configuration of a fluorescence observationsystem 1 according to a first embodiment of the present technology;

FIG. 2 is a view showing fluorescence characteristics of fluorescein;

FIG. 3 is a view showing spectral characteristics of a cobalt filter andlight components that passes through the cobalt filter and irradiatesthe specimen 17;

FIG. 4 is a view showing light components emitted from the specimen 17when the specimen 17 not dyed with fluorescein is irradiated with lightin a B-wavelength region of from about 350 nm to about 510 nm;

FIG. 5 is a view showing light components emitted from the specimen 17when the specimen 17 dyed with fluorescein is irradiated with the lightin the B-wavelength region of from about 350 nm to about 510 nm;

FIG. 6 is a view showing color filter spectral characteristics of a CMOSimage sensor 16;

FIG. 7 is a view showing a correction result of color components;

FIG. 8 is a view showing a configuration of the image processing unit 20shown in FIG. 1;

FIG. 9A is a view showing a setting example of a linear matrixtransformation coefficient;

FIG. 9B is a view showing another setting example of the linear matrixtransformation coefficient; and

FIG. 9C is a view showing still another setting example of the linearmatrix transformation coefficient.

DETAILED DESCRIPTION OF EMBODIMENTS

Hereinafter, an embodiment of the present technology will be describedwith reference to the drawings.

First Embodiment

FIG. 1 is a view showing a configuration of a fluorescence observationsystem 1 according to a first embodiment of the present technology.

This fluorescence observation system 1 includes an imaging unit 10 andan image processing unit 20. The imaging unit 10 and the imageprocessing unit 20 are connected to each other through an imagetransmission channel 30.

[Imaging Unit 10]

The imaging unit 10 includes a light source 11, an excitation filter 12,an objective lens 13, a beam splitter 14, an eye piece 15, a CMOS imagesensor 16, and the like.

The light source 11 is a halogen lump, a xenon arc lamp, or the like andemits white light including the wavelength band of visible light.

The excitation filter 12 is an optical filter that causes light having awavelength for exciting a fluorescent material of a specimen 17 to passtherethrough. The light (excited light) passing through the excitationfilter 12 irradiates the specimen 17.

The specimen 17 is a biological tissue slice, a cell group, or the likedyed with the fluorescent dye. Here, a case where the fluorescein isused as the fluorescent dye is assumed. As shown in FIG. 2, thefluorescein is fluorescent dye that emits green light having a centerwavelength of 520 nm when exposed to blue light having a centerwavelength of 490 nm.

FIG. 3 is a view showing spectral characteristics of a cobalt filterthat causes mainly light in a wavelength region of blue (B) to passtherethrough as the excitation filter 12 and light components thatpasses through the cobalt filter and irradiates the specimen 17. As canbe seen from the figure, the cobalt filter is an optical filter thatcauses light in a B-wavelength region of from about 350 nm to about 510nm. Here, B corresponds to a “first color” in the scope of claims. G andR correspond to a “second color” and a “third color” in the scope ofclaims.

Light emitted from the specimen 17 enters the objective lens 13. Thebeam splitter 14 is provided in a light path between the objective lens13 and the eye piece 15. That is, light from the objective lens 13 isdistributed into light directed to the eye piece 15 and light directedto the CMOS image sensor 16. With this, the same image as an observationimage that an observer views through the eye piece 15 can be captured.

Note that, although shown in the figure, an IR cut filter is provided ata preceding stage of the CMOS image sensor 16. The IR cut filter is anoptical filter for removing wavelength components equal to or largerthan 680 nm, for example, that is a wavelength region of red (R).

A complementary MOS (CMOS) image sensor 16 receives light passingthrough the IR cut filter for each of R, G, and B selected by an opticalfilter to generate an electrical signal and digitalizes it to generateRaw data. The generated Raw data is transmitted to the image processingunit 20 through the image transmission channel 30.

[Color Mixing in Fluorescence and Color Components of Background]

Next, color mixing in fluorescence, which is caused when the imagingunit 10 captures a fluorescence image, and color components of thebackground.

FIG. 4 is a view showing light components emitted from the specimen 17when light in the B-wavelength region of from about 350 nm to about 510nm limited by the excitation filter 12 irradiates the specimen 17 notdyed with fluorescein. At this time, light emitted from the specimen 17is mainly reflected light. Components of the reflected light is equal tocomponents of the light (FIG. 3) irradiating the specimen 17. The colorof an image captured at this time is dark blue color.

FIG. 5 is a view showing light components emitted from the specimen 17when light in the B-wavelength region of from about 350 nm to about 510nm irradiates the specimen 17 dyed with the fluorescein. In this case,by the specimen 17 died with the fluorescein being exposed to the lightin the B-wavelength region, the fluorescence is generated. In FIG. 5, G′indicates G-color components of the fluorescence.

FIG. 6 is a view showing color filter spectral characteristics of theCMOS image sensor 16.

As can be seen from the color filter spectral characteristics, G andR-pixels of the CMOS image sensor 16 react also to the light in theB-wavelength region of from about 350 nm to about 510 nm that is atransmission region of the cobalt filter. Thus, a signal detected by theCMOS image sensor 16 is the components of the fluorescence to whichcomponents (mixed components) generated when R, G, and B-pixels react tothe reflected light in the B-wavelength region of from about 350 nm toabout 510 nm.

In this manner, the CMOS image sensor 16 outputs a sum of G-componentsgenerated by reacting to the fluorescence and R, G, and B-componentsgenerated by reacting to the reflected light in the B-wavelength regionof from 350 nm to about 510 nm. Thus, it is not possible to obtain thetrue color of the fluorescence image and obtain a sufficient colorcontrast between a background portion and the fluorescence image becausethe background portion is dark blue. The present technology has beenmade for at least improving this point.

Hereinafter, details thereof will be described.

With respect to the light in a G-wavelength region of from about 350 nmto about 510 nm, the R, G, and B-components detected by the CMOS imagesensor 16 are expressed as follows.

[Formula 1]

R=∫ _(a) ^(b) {E*HG(x)*CB(x)*Dr(x)}  (1)

G=∫ _(a) ^(b) {E*HG(x)*CB(x)*Dg(x)}  (2)

B=∫ _(a) ^(b) {E*HG(x)*CB(x)*Db(x)}  (3)

Where E denotes energy (constant) of the light source 11, HG(x) denotescharacteristics (spectral distribution) indicative of energydistribution state of the light source 11, CB(x) denotes the spectralcharacteristics of the cobalt filter, Db(x), Dg(x), and Dr(x) denote B,G, and R-color filter spectral characteristics of the CMOS image sensor16, respectively, and a and b denote the transmission region of thecobalt filter (G-wavelength region of from about 350 nm to about 510nm).

Ng % is a percentage of components detected by the G-pixels of the CMOSimage sensor 16 with respect to components detected by the B-pixels inthe G-wavelength region of from about 350 nm to about 510 nm. That is,it is expressed by Ng %=G/B.

Nr % is a percentage of components detected by the R-pixels of the CMOSimage sensor 16 with respect to the components detected by the B-pixelsin the wavelength region limited by the cobalt filter. That is, it isexpressed by Nr %=R/B.

As described above, components detected by the CMOS image sensor 16 whenthe specimen 17 dyed with the fluorescein is irradiated with the lightin the G-wavelength region of from about 350 nm to about 510 nm are asum of the G-components generated by reacting to the fluorescence andthe R, G, and B-components generated by reacting to the reflected lightin the B-wavelength region of from 350 nm to about 510 nm.

Thus, if G=B*Ng %, R=B*Nr %, and the B-components are removed from theall R, G, and B-components detected by the CMOS image sensor, only colorcomponents G′ of G generated by reacting to the fluorescence remain asshown in FIG. 7.

By adjusting the color components as described above, the true G-colorof the fluorescence image can be obtained. Further, the backgroundportion is closer to the true black because the color components areremoved, and the color contrast between the background and thefluorescence portion is improved.

Next, a circuit that adjusts the above-mentioned color components fromthe output of the CMOS image sensor 16 will be described.

[Configuration of Image Processing Unit 20]

FIG. 8 is a view showing a configuration of the image processing unit20.

The Raw data that is the output of the CMOS image sensor 16 is input inthe image processing unit 20 through a sensor I/F 21.

The Raw data input through the sensor I/F 21 is introduced into a whitebalance adjustment circuit 22.

The white balance of the image is adjusted at the white balanceadjustment circuit 22. An optical interface circuit (OP I/F) 23 suppliesthe output of the white balance adjustment circuit 22 to an EVcorrection circuit 24. The OP I/F 23 interpolates the color, generatesthe R, G, and B-color signals, and further generates a luminance signal(Y) for edge processing from the G-signal, for example. The OP I/F 23introduces the R, G, and B-color signals and the luminance signal (Y)into a separation circuit 25. The EV correction circuit 24 is a circuitthat adjusts the exposure of the CMOS image sensor 16 for adjusting thebrightness of the image.

The separation circuit 25 introduces the R, G, and B-color signals intoa color correction circuit 26 and introduces the luminance signal (Y)into an edge processing circuit 27. At the edge processing circuit 27,edge(s) is detected from the luminance signal (Y), components of thedetected edge are adjusted, and so on.

The color correction circuit 26 is a circuit that adjusts theabove-mentioned R, G, and B-color components by linear matrixtransformation. In the linear matrix transformation, the colorcomponents are adjusted by matrix operation on the R, G, and B-colorsignals.

The output of the color correction circuit 26 is introduced into a gammacorrection circuit 28, subjected to gamma correction there, andconverted by a YC conversion circuit 29 into the luminance signal (Y)and a color difference signal (C). The luminance signal (Y) is added tothe edge components introduced from the edge processing circuit 27 andbecomes the luminance signal (Y) in which the edge is emphasized. Theedge-emphasized luminance signal (Y) is subjected to a process such asnoise removal by a YC processing circuit 31 and output. On the otherhand, the color difference signal (C) is also subjected to a processsuch as noise removal by the YC processing circuit 31 and output.

Note that the image processing unit 20 is specifically constituted ofone large scale integration (LSI) or a plurality of LSIs, ICs, and thelike.

[Color Adjustment of Linear Matrix Transformation]

A matrix operation formula used in the linear matrix transformation isshown.

$\begin{matrix}\left\lbrack {{Fomula}\mspace{14mu} 2} \right\rbrack & \; \\{\begin{pmatrix}{Rout} \\{Gout} \\{Bout}\end{pmatrix} = {\begin{pmatrix}{RR} & {RG} & {RB} \\{GR} & {GG} & {GB} \\{BR} & {BG} & {BB}\end{pmatrix}\begin{pmatrix}{Rin} \\{Gin} \\{Bin}\end{pmatrix}}} & (4)\end{matrix}$

Here, out of coefficients RR, RG, RB, GR, GG, GB, BR, BG, and BB of nineelements in the matrix, RR, GG, and BB are automatically calculated suchthat a sum of values of three coefficient in a row to which each of thembelongs is “1”. Thus, the coefficients that should be actually selectedare the following six: RG, RB, GR, GB, BR, and BG.

In order to select the above-mentioned six coefficients, values ofR=B*Nr %, G=B*Ng %, and the B-components derived from color filterspectral characteristics of the CMOS image sensor 16 are used as aninput. At this time, the above-mentioned coefficients are selected suchthat the output of each of R, G, and B is closer to “0.” That is, thecoefficients are selected such that the following formula is satisfied.

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 3} \right\rbrack & \; \\{\begin{pmatrix}{Rout} \\{Gout} \\{Bout}\end{pmatrix} = {{\begin{pmatrix}\left( {1 - {RG} - {RB}} \right) & {RG} & {RB} \\{GR} & \left( {1 - {GR} - {GB}} \right) & {GB} \\{BR} & {BG} & \left( {1 - {BR} - {BG}} \right)\end{pmatrix}\begin{pmatrix}{{Bin}*{Nr}\%} \\{{Bin}*{Ng}\%} \\{Bin}\end{pmatrix}\,} \approx \begin{pmatrix}0 \\0 \\0\end{pmatrix}}} & (5)\end{matrix}$

In actual operation, the color correction circuit 26 in which thecoefficients are selected in the above-mentioned manner is used toperform the color adjustment with respect to each of the input R, G, andB-components.

[Specific Coefficient Setting Examples]

FIGS. 9A, 9B, and 9C show specific coefficient setting examples.

FIG. 9A shows a setting example in which the output of each of R, G, andB was closest to “0” among the three setting examples, that is, atheoretically optimal setting example. The effect of the suppression ofcolor mixing and improvement of the contrast was as expected. However,in a low-luminance environment or with a large negative coefficientvalue, the noise amount can be increased.

FIG. 9B is a setting example in which the coefficient values of GR, GG,GB, BR, and BB were changed for reducing the noise. In this setting, thenegative coefficient value was decreased, and hence the effect of thesuppression of color mixing and improvement of the contrast was slightlyreduced but the effect of reduction of the noise amount has beenrecognized.

FIG. 9C is a setting example in which the coefficients of RR, RB, GG,and GB were further changed from the setting example shown in FIG. 9B.In this setting, the negative coefficient value was further decreased,and hence the effect of the suppression of color mixing and improvementof the contrast was further reduced but a sufficient effect of reductionof the noise amount has been recognized.

Note that, in accordance with the relationship among RR+RG+RB=1,GR+GG+GB=1, and BR+BG+BB=1, if one coefficient in each row of the matrixis determined, another coefficient is also determined according to thefollowing formulae (6), (7), and (8).

(1−RG−RB)*Nr %+RG*Ng %+RB=0

Nr %+(Ng %−Nr %)*RG+(1−Nr %)*RB=0

RG=−((1−Nr %)*RB−Nr %)/(Ng %−Nr %)  (6)

Nr %*GR+Ng %−Ng %*GR−Ng %*GB+GB=0

(Nr %−Ng %)*GR+(1−Ng %)*GB+Ng %=0

GR=((Ng %−1)*GB−Ng %)/(Nr %−Ng %)  (7)

Nr %*BR+Ng %*BG+1−BR−BG)=0

(Nr %−1)*BR+(Ng %−1)*BG+1=0

BR=(−1−(Ng %−1)*BG)/(Nr %−1)  (8)

Although the case where the constraints of RR+RG+RB=1, GR+GG+GB=1, andBR+BG+BB=1 are present has been described in the above, the presenttechnology is not limited thereto. If these constraints are not present,it is possible to select the coefficient values with a higher degrees offreedom.

[Reduction of R-Components at White Balance Adjustment Circuit]

Although the above-mentioned color correction is performed only by thelinear matrix transformation at the color correction circuit 26, thecolor correction may be performed partially at the white balanceadjustment circuit.

For example, the R-components may be reduced as much as possible at thewhite balance adjustment circuit 22. In this case, the load on thecoefficients set in the color correction circuit 26 can be reduced, andhence an effect of reducing the noise is provided. In other words, thecalculated amount of noise is increased if the coefficients takenegative large values in the linear matrix transformation. By setting asignal from which the R-components are removed at the white balanceadjustment circuit 22 as a target of the linear matrix transformation,it is possible to prevent the coefficients form taking the negativelarge values and to reduce the amount of noise.

As described above, in the image processing unit 20 according to thisembodiment, by the color correction circuit 26 or the like adjusting theR, G, and B-color components, the components (mixed color components)generated when the R, G, and B-pixels of the CMOS image sensor 16 reactto the reflected light in the B-wavelength region that is used as theexcitation light are removed. With this, the true G-color of thefluorescence image can be obtained. The background portion is closer tothe true black because the color components are removed, and the colorcontrast between the background and the fluorescence portion isimproved.

Further, the mixed color components can be removed by the process of thecolor correction circuit 26 or the like within the image processing unit20 in the above-mentioned manner, and hence it becomes unnecessary toplace an absorption filter for cutting the reflected light in theB-wavelength region in the imaging unit 10. The absorption filter in theimaging unit 10 becomes unnecessary, and hence it is possible to reducethe size, weight, and cost of the imaging unit 10. In addition, aneffect that the current imaging apparatus (slit lamp) can be used as itis can be also provided.

Modified Example 1

In each of the above-mentioned embodiments, the case where thefluorescein as the fluorescent dye is used is assumed. However, thepresent technology is applicable also to the case where otherfluorescent dye is employed.

Modified Example 2

In each of the above-mentioned embodiments, by the linear matrixtransformation at the color correction circuit 26, G=B*Ng %, R=B*Nr %,and the B-components are removed from the all R, G, and B-componentsdetected by the CMOS image sensor 16.

As a modification thereof, by the linear matrix transformation at thecolor correction circuit 26, only G=B*Ng % and R=B*Nr % are removed fromthe all R, G, and B-components detected by the CMOS image sensor 16.Regarding the B-components, a circuit for removing the B-components maybe provided at a subsequent stage of the color correction circuit 26 andthe B-components may be removed there.

Note that the present technology may also take the followingconfigurations.

(1) An image processing apparatus, including:

an interface unit configured to input an image signal from an imagingapparatus that exposes a specimen dyed with a fluorescent dye toexcitation light and images fluorescence by a color imaging element; and

a color correction circuit configured to retain information on apercentage of each of a component of a second color and a component of athird color with respect to a component of a first color correspondingto the excitation light in the image signal, which is determined inadvance based on color filter spectral characteristics of the colorimaging element, and reduce each of an amount corresponding to thepercentage of the component of the second color and an amountcorresponding to the percentage of the component of the third color fromthe input image signal.

(2) The image processing apparatus according to (1), in which

the color correction circuit is configured to reduce the component ofthe first color from the input image signal.

(3) The image processing apparatus according to (1) or (2), in which

the color correction circuit is a linear matrix transformation circuit.

(4) The image processing apparatus according to any one of (1) to (3),further including

a white balance adjustment circuit configured to reduce either one ofthe component of the second color and the component of the third colorfrom the input image signal at a preceding stage of the color correctioncircuit.

(5) The image processing apparatus according to any one of (1) to (4),in which

the fluorescent dye is fluorescein.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

What is claimed is:
 1. An image processing apparatus, comprising: aninterface unit configured to input an image signal from an imagingapparatus that exposes a specimen dyed with a fluorescent dye toexcitation light and images fluorescence by a color imaging element; anda color correction circuit configured to retain information on apercentage of each of a component of a second color and a component of athird color with respect to a component of a first color correspondingto the excitation light in the image signal, which is determined inadvance based on color filter spectral characteristics of the colorimaging element, and reduce each of an amount corresponding to thepercentage of the component of the second color and an amountcorresponding to the percentage of the component of the third color fromthe input image signal.
 2. The image processing apparatus according toclaim 1, wherein the color correction circuit is configured to reducethe component of the first color from the input image signal.
 3. Theimage processing apparatus according to claim 2, wherein the colorcorrection circuit is a linear matrix transformation circuit.
 4. Theimage processing apparatus according to claim 3, further comprising awhite balance adjustment circuit configured to reduce either one of thecomponent of the second color and the component of the third color fromthe input image signal at a preceding stage of the color correctioncircuit.
 5. The image processing apparatus according to claim 4, whereinthe fluorescent dye is fluorescein.
 6. An image processing method,comprising: determining, based on color filter spectral characteristicsof a color imaging element that exposes a specimen dyed with afluorescent dye to excitation light and images fluorescence, apercentage of each of a component of a second color and a component of athird color with respect to a component of a first color correspondingto the excitation light in an image signal of the color imaging element;and reducing each of an amount corresponding to the percentage of thecomponent of the second color and an amount corresponding to thepercentage of the component of the third color from the image signalinput from the color imaging element.